I'd play a 5k giant army with my elves, orcs or other army without any real concern.
I'll give you a run down of why rather than just making grandiose statements.
Firstly their magic is abismal. Magic 2 will save them from nothing. Magic missile's will take them apart quickly. Spells affecting movement will destroy any ordered battle line/plan. Their own magicians are poor having only general spells as options which a next to worthless as they make only marginal difference to already poor skills or slight boosts to their strengths. They are also too costly to take in numbers as they can't be spread widely enough to benefit enough units with their magic save. They also have the reduced combat abilities of a young giant. And i know those abilities aren't poor but it's a reduction in power that makes them less worth taking for the points outlay.
Next - their missiles are nothing but a nuisance. The thrown rocks will cause at best one kill per turn per unit as they will need to move their full movement each turn to get their combat power into play. Their lobbas and balista wont be taken as they cant move and shoot which again takes away their combat strength which brings us onto their main strength combat with rollers.
In 5000 points your probably going to face a mage and two marshals at 1100 points approx. You then put rollers on the young giants to add to their lesser combat power. Units of three are the norm but four can be taken. So that 600 per unit. Taking two gives you 1200 points. Taking units of three adults to go with each of the three characters gives 2160
this gives a total of nearly 4500. This gives some room for another mage or some monsters as theres some space left in the elites section if not a lot which reminds me of something i missed earlier which is you rarely get room for monsters given how expensive your characters are.
So giants will have about 5- 6 units on average, more if you break them up into units of twos which are pointless as one of them killed results in a combat ineffective unit. This is pretty small compared to most armies at 5k points and more than counters the fact that the giants break the CCF elite rule.
I don't see this or any other combo of unit formation being difficult to beat. They are moderately quick at 8 inches though not really that fast. Rollers reduce them to 6 and therefore either breaks up a battle line or makes the other units move at 6 too. Concentrated fire and magic will punch holes in their battle line quickly. Tactical manouevring and withdrawals will add turns for shooting as will tempting the giants general with tasty throwaway units or things that can flee and rally quickly and draw off the few giant units. Once holes have been created rush your fast units through them or your monsters and start picking at the giants from behind. Then gang your units up on the giant from the flank or rear and in skirmish thus negating or massively minimising rams and sweeps. Whilst benefitting from from long weapons against the rollers.
I'll stop there and i'm sorry to sound a bit like a lecturer. Whilst i agree that not all the lists are completely balanced I cannot accept that the giants are overpowered. They have some great strengths but plenty of exploitable weaknesses. A good general just has to search them out and use them. I personally love to face them as they give different challenges to the other armies. If we grey them down then we risk losing the variety that makes this a great game. I most certainly can't see a good reason to change them.
Anyway it's the halflings that scare me. Whenever Rocket toad decides to give them an outing again