Boeing could have chosen to compete with the A380 and built their own Superjumbo, but they chose not to, instead working on the Dreamliner.
The 744, which was the last update to the 747 to see large market penetration, wasn't all that cost effective either:
> with 70 percent of its seats occupied, used more than 95 percent of the fuel needed by a fully occupied 747
Yes, the A380 is more fuel efficient, and cost per seat is much lower, but I suspect with the exception of Emirates, most airlines did not need bigger planes. Other airlines such as BA, Quantas and Singapore bought A380s to replace their oldest 744s, but still have a lot of 744s in service that won't be replaced with A380s.
The 744, which was the last update to the 747 to see large market penetration, wasn't all that cost effective either:
> with 70 percent of its seats occupied, used more than 95 percent of the fuel needed by a fully occupied 747
Yes, the A380 is more fuel efficient, and cost per seat is much lower, but I suspect with the exception of Emirates, most airlines did not need bigger planes. Other airlines such as BA, Quantas and Singapore bought A380s to replace their oldest 744s, but still have a lot of 744s in service that won't be replaced with A380s.