Don't know much about waf, but CMake fits your requirements pretty well. I do know waf is written in Python, my personal favourite programming language ATM.
My primary requirements are:
Must be able to compile software in windows/linux and android.
CMake does Windows and Linux very well but so does any other build system worth its salt. Someone wrote some Android scripts for CMake.
Must be ready to run tests.
CMake has a sibling testing framework.
Must be able to play nicely with other libraries that must be compiled with another build system but most likely will have to be compiled from source.
CMake has good integration with pkg-config, and can link against arbitrary shared libraries.
Must be able to add custom steps, like for example, generating some data from some files (mainly graphics) before compiling, all integrated in the build system.
CMake can generate custom rules, but the Makefile backends have the best support for this (ie. custom make targets.) You can always run arbitrary commands from cmake itself, though, with any backend.
Some strong preferences are:
Being ready to support MAC compilation.
CMake supports Mac quite well. It will even make you an Xcode project if you want, but it can also do command line builds.
Being able to cross-compile from linux as many platforms as I can (maybe windows/linux/android but cannot MAC?)
Cross-compiling is supported in CMake. CMake will not be the primary source of pain with cross-compiling - literally everything else will.
Especially with regards to cross-compiling for Mac. It's possible, but not worth it to cross-compile for that platform, considering you need access to a Mac anyways to get the libraries and header files, you need to patch GCC and clang and LLVM, etc. The only sound reason I've heard for going through this much pain is running an automated build server. Anyways, if you get a working Linux -> Mac toolchain, you should be able to cross-compile with CMake as if it were any other Unix platform.
Being able to add support for iOS compilation if the need arises.
iOS cross-compilation is supported by CMake, but you need a Mac.
Would be nice if the invocation interface was similar to that of autotools, since it is the one many people know and it is well documented.
Write a configure script that just calls CMake. Then your users can do a ./configure && make
. There's also CPack which lets you generate DEB, RPM, NSIS (Windows) and DMG (Mac) installers/packages.
Some questions:
If I have some rare requirement, which build system would be more ready to be extended?
CMake is very extensible. It can be extended to support new languages and target platforms. (Given that waf is written in Python, it's going to be pretty hackable too.)
Are both currently well maintained? (I wonder about waf mainly).
CMake is mature and well-maintained.
Community: if I find a problem, both communities are big enough to support me, in your experience?
The community and extensions available are what keeps me coming back to CMake, honestly.