If my Wcf Service and Web Application, both are in same server and if i want to access my web app over internet means which WCF security i have to use and why ?
Please advise me :)
It depends on binding and the context usage and not on transactions which is a different topic.
The intranet bindings (NetTcpBinding, NetNamedPipeBinding, and NetMsmqBinding) all default to Transport security. Thus, no special programming is required on behalf of the service or client developer. The reason is that on the intranet calls are typically point-to-point, and Transport security yields the best performance. However, the intranet bindings can also be configured for the None transfer mode; that is, they can be used on the same transport protocol, only without security. The NetNamedPipeBinding supports only None and Transport security—there is no sense in using Message security over IPC, since with IPC there is always exactly one hop from the client to the service. Also note that only the NetMsmqBinding supports the Both mode.
The Internet bindings all default to Message security, to enable them to be used over nonsecure transports (that is, HTTP) and to accommodate multiple hops and intermediaries.