What was a major issue with building the atomic bomb?

What was a major issue with building the atomic bomb?