In general, Usability Engineering encompasses all of the parts of designing an interfaces in a way that users will be able to learn, comprehend, and remember easily.
When you're developing a software library, you are designing an interface. In fact, it's an Application Programming Interface, or API, which your users - other developers - will need to learn, comprehend, and remember easily.
API Usability is the intersection of good usability design and good programming techniques. Below, I discuss some of the things that I think are critical for a usable API.
Begin with the end in mind. Ask yourself what you want to accomplish with your API. Develop some sample applications that will use your API before you even start with the API itself. This will help you refine the methods that you intend to define. This guidance is also the key of Test-Driven Development.
Make conceptually easy things simple to do. If all a developer wants to do is make their computer beep, they shouldn't need to open a sound data line, create a clip object, generate bytes for the frequencies, send the clip to the data line, and on and on. The should be able to say, "BEEP(freq)".
Create a compact API. The fewer lines of code the user has to write, the better! Be especially aware of code that you repeat over and over in your example cases - this indicates boilerplate that you can probably put into your API.
Be absolutely correct. I borrowed this from Elliotte Rusty Harold's XOM Design Principles. He's right: people are relying on your API to do what it says it will do, so make sure it works correctly. It must also generate correct output.
Be available for comments and bugs. This is a feedback process that will enhance your API. It's always good to have extra sets of eyes looking at your work.
Construct complete objects only. Don't allow a user to construct an object that requires setting critical properties later - first, they might not call those setters; second, there's nothing in the build environment that will force the user to call your methods.
Catch errors right away. If something goes wrong, report it! No sense in going down a path that won't work in the end.
Be verbose in reporting errors. "Error! Program stopping" is a really bad error. "The file 'filename.ext' was not found in directory 'directory'" is much better.
Be open in communicating an evolving API. If people have older versions of your API and you release a newer version, they need to know what changed. Also, realize that it's pretty difficult to change an API once it's released, since many people will have code that relies on it. This is why Java has so many deprecated methods that have not been removed (yet).
The success of your API project also depends on your presentation. If someone goes to your webpage and can't figure out what your API does, how to download it, how to use it, and what some simple examples might be, they probably won't use it.