I need a very simple server that listens for connections, retrieves a question and simply answers yes or no. Then the connection is broken.
As my Android device is the only device that is powered on and connected to the internet 24/7, I'm thinking of using it as the server.
Is this possible, and how could I achieve this? The biggest problems that I'm facing is the ip-address, as it could be changing a lot. I'm thinking of storing it in a public dropbox file, so the client can retrieve the address from that
There's a whole slew of reasons why you shouldn't use your android device as a server, but I won't go into those. I'll just assume you have a good reason for doing what you want to do.
Since your question is very general, the best I can do is give you a very general answer. You'd implement a server on android like you would any other server using java using sockets. For a simple java implementation of a socket based server, checkout this tutorial. The tricky part is going to be getting your IP address to share with clients so that they can connect to your device. As nick suggested, dyndns might be an option here, but I'm not sure if they have a mobile client.