Usually we use square brackets - [ ] - for special purposes such as in math
For example : y(x)=[(x^2+3)^3+3]^5
Round brackets - ( ) - are used in a similar way to commas when we want to add further explanation, an afterthought, or comment that is to do with our main line of thought but distinct from it.
We use this when we are solving math too
for example: 5(4)(4)
We use curly brackets {} when there are equal choice on something
For example:
Select your pizza topping {pepper, onion, sausage, tomato, feta, anchovies, bacon, sun-dried tomatoes, chicken, broccoli}.
or,
When you are a mathematician you might use this if you are solving a problem like this
()
or,
when you are programming
{text2}
It can be easier to understand the line of writing.