Analyze the principles of HTTPS and its use in Android

1. Shortcomings of HTTP protocol

When HTTP1.x transmits data, all transmitted content is in clear text. Neither the client nor the server can verify the identity of the other party. The existing problems are as follows:

• Communication uses clear text (not encrypted), and the content may be eavesdropped;
• If the identity of the communicating party is not verified, it may be disguised;
• The integrity of the message cannot be proven, so it may have been tampered with;

In fact, these problems not only occur in HTTP, but also in other unencrypted protocols.

(1) Communication using clear text may be eavesdropped

According to the working mechanism of the TCP/IP protocol suite, there is a risk of communication content being eavesdropped in any corner of the Internet. The HTTP protocol itself does not have the encryption function, and all data transmitted are plain text. Even if the communication has been encrypted, the communication content will be peeked into, which is the same as the unencrypted communication. It just means that if the communication is encrypted, it may be impossible to decipher the meaning of the message information, but the encrypted message itself will still be seen.

(2) Failure to verify the identity of the communicating party may lead to disguise

During HTTP protocol communication, since there are no processing steps to confirm the communicating party, anyone can initiate a request. In addition, as long as the server receives the request, it will return a response regardless of who the other party is. Therefore, if the communication party is not confirmed, there are the following hidden dangers:

• It is impossible to determine whether the web server that the request is sent to is the one that returns the response according to the true intention. It may be a disguised web server;
• It is impossible to determine whether the client the response is returned to is the client that received the response according to the true intention. It may be a disguised client;
• Unable to determine whether the other party being communicated has access rights. Because some web servers store important information, they only want to grant communication permissions to specific users;
• It is impossible to determine where the request came from and who made it;
• Even meaningless requests will be accepted as ordered, unable to prevent DoS attacks under massive requests;

(3) The integrity of the message cannot be proven and may have been tampered

The so-called integrity refers to the accuracy of information. Failure to demonstrate its completeness usually means that it cannot be judged whether the information is accurate. The HTTP protocol cannot prove the integrity of the communication messages. During the period after the request or response is sent until the other party receives it, there is no way to know even if the content of the request or response has been tampered with.

For example, when downloading content from a certain Web website, it is impossible to determine whether the files downloaded by the client and the files stored on the server are consistent. The file content may have been tampered with to other content during transmission. Even if the content has really changed, the client as the recipient is not aware of it. Like this, an attack in which the request or response is intercepted by the attacker and tampered with the content during transmission is called a Man-in-the-Middle attack (MITM).

(4) Several characteristics that a secure HTTP version should have

Due to the above problems, an HTTP security technology that can provide the following functions is needed:

(1) Server authentication (clients know they are talking to the real server and not a fake server);

(2) Client authentication (servers know they are talking to a real client and not a fake client);

(3) Integrity (client and server data will not be modified);

(4) Encryption (the conversation between the client and the server is private, no need to worry about being eavesdropped);

(5) Efficiency (an algorithm that runs fast enough to be used by low-end clients and servers);

(6) Universality (basically all clients and servers support these protocols);

2. Key technologies of HTTPS

In the context of such demand, HTTPS technology was born. The main functions of the HTTPS protocol basically rely on the TLS/SSL protocol, which provides authentication, information encryption and integrity verification functions, which can solve the security problems of HTTP. This section will focus on several key technical points of the HTTPS protocol.

(1) Encryption technology

Encryption algorithms are generally divided into two types:

Symmetric encryption: The keys for encryption and decryption are the same. Represented by DES algorithm;

Asymmetric encryption: The keys for encryption and decryption are different. Represented by RSA algorithm;

Symmetric encryption is very strong and generally cannot be cracked. However, there is a big problem that the key cannot be generated and stored securely. If each session between the client and the server is encrypted with a fixed, same key. and decryption, there must be great security risks.

Before the emergence of asymmetric key exchange algorithms, a big problem with symmetric encryption was not knowing how to safely generate and store keys. The asymmetric key exchange process is mainly to solve this problem and make the generation and use of keys more secure. But it is also the “culprit” that severely reduces HTTPS performance and speed.

HTTPS uses a hybrid encryption mechanism that uses both symmetric encryption and asymmetric encryption. It uses asymmetric encryption in the key exchange process, and uses symmetric encryption in the subsequent communication and message exchange stages.

(2) Authentication – Certificate proving the correctness of the public key

One of the biggest problems with asymmetric encryption is that it cannot prove that the public key itself is the genuine public key. For example, when preparing to establish communication with a certain server using public key encryption, how to prove that the public key received is the public key issued by the originally expected server. Perhaps during the public key transmission, the real public key has been replaced by the attacker.

If the reliability of the public key is not verified, there will be at least the following two problems: man-in-the-middle attack and information denial.

In order to solve the above problems, you can use public key certificates issued by the digital certificate certification authority (CA, Certificate Authority) and its related agencies.

The specific process for using CA is as follows:

(1) The server operator applies for a public key to the digital certificate certification authority (CA);

(2) CA verifies the authenticity of the information provided by the applicant through online, offline and other means, such as whether the organization exists, whether the enterprise is legal, whether it has ownership of the domain name, etc.;

(3) If the information is reviewed and passed, the CA will digitally sign the applied public key, then distribute the signed public key, and put the public key into the public key certificate and bind them together. . The certificate contains the following information: the applicant's public key, the applicant's organizational information and personal information, the information of the issuing authority CA, the validity time, the certificate serial number and other information in plain text, and also contains a signature; the signature generation algorithm: first, use hash The column function calculates the information digest of the public plaintext information, and then uses the CA's private key to encrypt the information digest, and the ciphertext is the signature;

(4) The client sends a request to the server during the HTTPS handshake phase, asking the server to return the certificate file;

(5) The client reads the relevant plaintext information in the certificate, uses the same hash function to calculate the information digest, then uses the public key of the corresponding CA to decrypt the signature data, and compares the information digest of the certificate. If they are consistent, Then you can confirm the legitimacy of the certificate, that is, the public key is legal;

(6) The client then verifies the domain name information, validity time and other information related to the certificate;

(7) The client will have built-in trust CA certificate information (including public key). If the CA is not trusted, the certificate corresponding to the CA will not be found, and the certificate will also be determined to be illegal.

Pay attention to a few points during this process:

(1) There is no need to provide a private key when applying for a certificate, ensuring that the private key can only be mastered by the server;

(2) The validity of the certificate still depends on the asymmetric encryption algorithm. The certificate mainly adds server information and signature;

(3) The certificate corresponding to the built-in CA is called the root certificate; the issuer and the user are the same, and they sign for themselves, which is called a self-signed certificate;

(4) Certificate = Public Key Applicant and Issuer Information Signature;

3.HTTPS protocol principle

(1) History of HTTPS

History of HTTPS protocol:

• (1) The first version of the SSL protocol was developed by Netscape, but this version was never released;
• (2) The second version of the SSL protocol was released in November 1994. The first deployment was on the Netscape Navigator 1.1 browser, released in March 1995;
• (3) SSL 3 was released in late 1995. Although the name is the same as the earlier protocol version, SSL3 is a completely redesigned protocol, and the design is still used today.
• (4) TLS 1.0 came out in January 1999. Compared with SSL 3, the version modifications are not major;
• (5) In April 2006, the next version TLS 1.1 was released, which only fixed some key security issues;
• (6) In August 2008, TLS1.2 was released. This version adds support for authenticated encryption and essentially removes all hard-coded security primitives from the protocol description, making the protocol fully resilient;

(2) Protocol implementation

Macroscopically, TLS is implemented by the record protocol. The record protocol is responsible for exchanging all low-level messages over the transport connection and can be configured for encryption. Every TLS record starts with a short header. The header contains the type (or subprotocol), protocol version, and length of the record's contents. The message data follows the header, as shown in the following figure:

The TLS master specification defines four core sub-protocols:

• Handshake protocol;
• change cipher spec protocol;
• Application data protocol;
• Alert protocol;

(3) Handshake protocol

The handshake is the most sophisticated part of the TLS protocol. During this process, the communicating parties negotiate connection parameters and complete identity authentication. Depending on the functions used, the entire process usually requires the exchange of 6 to 10 messages. There may be many variations of the exchange process depending on the configuration and supported protocol extensions. The following three processes can often be observed in use:

• (1) Complete handshake, authenticating the server (one-way authentication, the most common);
• (2) Handshake to authenticate both client and server (two-way authentication);
• (3) Brief handshake used to restore the previous session;

(4) One-way verification handshake process

This section takes the login process of QQ mailbox as an example to analyze the handshake process of one-way verification by capturing packets. A complete handshake process for one-way verification is as follows:

Mainly divided into four steps:

• (1) Exchange the functions supported by each and agree on the required connection parameters;
• (2) Verify the certificate presented, or use other methods for authentication;
• (3) Agree on the shared master key that will be used to protect the session;
• (4) Verify whether the handshake message has been modified by a third party;

The following is a detailed analysis of this process.

1.ClientHello

In the handshake process, ClientHello is the first message. This message conveys the client's capabilities and preferences to the server. Contains the specified version of SSL supported by the client and a Cipher Suite list (encryption algorithm used, key length, etc.).

2.ServerHello

The ServerHello message transmits the connection parameters selected by the server back to the client. The structure of this message is similar to ClientHello, except that each field contains only one option. The server's encryption component content and compression method are all filtered out from the received client encryption components.

3.Certificate

The server then sends a Certificate message, which contains a public key certificate. The server must ensure that the certificate it sends is consistent with the selected algorithm suite. However, the Certificate message is optional because not all packages use authentication, and not all authentication methods require certificates.

4.ServerKeyExchange

The purpose of the ServerKeyExchange message is to carry additional data for key exchange. Message content will differ for different negotiation algorithm suites. In some scenarios, the server does not need to send anything, and in these scenarios there is no need to send a ServerKeyExchange message.

5.ServerHelloDone

The ServerHelloDone message indicates that the server has completed sending all expected handshake messages. After this, the server waits for the client to send the message.

6.ClientKeyExchange

ClientKeyExchange message carries all information provided by the client for key exchange. This message is affected by the negotiated cipher suite, and the content varies with the different negotiated cipher suites.

7.ChangeCipherSpec

The ChangeCipherSpec message indicates that the sender has obtained sufficient information to generate connection parameters, has generated an encryption key, and will switch to encryption mode. Both the client and the server will send this message when conditions are ripe. Note: ChangeCipherSpec does not belong to the handshake message. It is another protocol with only one message, implemented as its sub-protocol.

8.Finished

Finished message means the handshake has been completed. The message content will be encrypted so that both parties can securely exchange the data needed to verify the integrity of the entire handshake. Both the client and the server will send this message when conditions are ripe.

(5) Handshake process of two-way verification

In some scenarios with higher security requirements, there may be a need for two-way verification. The complete two-way verification process is as follows:

It can be seen that compared with the one-way verification process, two-way verification has the following two additional messages: CertificateRequest and CertificateVerify. The rest of the process is roughly the same.

1.Certificate Request

Certificate Request is an optional feature specified by TLS and is used by the server to authenticate the client's identity. Implemented by the server asking the client to send a certificate, the server should send a CertificateRequest message immediately after ServerKeyExchange.

The message structure is as follows:

enum { 
    rsa_sign(1), dss_sign(2), rsa_fixed_dh(3),dss_fixed_dh(4), 
    rsa_ephemeral_dh_RESERVED(5),dss_ephemeral_dh_RESERVED(6), 
    fortezza_dms_RESERVED(20), 
    ecdsa_sign(64), rsa_fixed_ecdh(65), 
    ecdsa_fixed_ecdh(66),  
    (255) 
} ClientCertificateType; 

opaque DistinguishedName<1..2^16-1>;struct { 
    ClientCertificateType certificate_types<1..2^8-1>; 
    SignatureAndHashAlgorithm 
      supported_signature_algorithms<2^16-1>; 
    DistinguishedName certificate_authorities<0..2^16-1>; 
} CertificateRequest;

You can choose to send a list of certificate authorities that you accept, represented by their distinguished names.

2.CertificateVerify

When client authentication is required, the client sends a CertificateVerify message to prove that it indeed possesses the private key of the client certificate. This message is only sent if the client certificate has signing capabilities. CertificateVerify must immediately follow ClientKeyExchange. The message structure is as follows:

struct {  
Signature handshake_messages_signature;  
} CertificateVerify;

(6) Application data protocol

The application data protocol carries application messages. From the perspective of TLS, these are the data buffers. The records layer packages, defragments, and encrypts these messages using the current connection security parameters. As shown in the figure below, you can see that the transmitted data has been encrypted.

(7) alert protocol

The purpose of the alarm is to notify the peer of abnormal communication conditions through a simple notification mechanism. It usually carries a close_notify exception, which is used when the connection is closed to report errors. Alerts are very simple, with only two fields:

struct {  
    AlertLevel level;  
    AlertDescription description;  
} Alert;

• AlertLevel field: Indicates the severity of the alert;
• AlertDescription: Directly represents the alert code;

4. Frequently asked questions about using HTTPS in Android

(1) 服务器证书验证错误

这是最常见的一种问题，通常会抛出如下类型的异常：

出现此类错误通常可能由以下的三种原因导致：

• (1) 颁发服务器证书的CA未知;
• (2) 服务器证书不是由CA签署的，而是自签署(比较常见);
• (3) 服务器配置缺少中间 CA;

当服务器的CA不被系统信任时，就会发生 SSLHandshakeException。可能是购买的CA证书比较新，Android系统还未信任，也可能是服务器使用的是自签名证书(这个在测试阶段经常遇到)。

解决此类问题常见的做法是：指定HttpsURLConnection信任特定的CA集合。在本文的第5部分代码实现模块，会详细的讲解如何让Android应用信任自签名证书集合或者跳过证书校验的环节。

(2) 域名验证失败

SSL连接有两个关键环节。首先是验证证书是否来自值得信任的来源，其次确保正在通信的服务器提供正确的证书。如果没有提供，通常会看到类似于下面的错误：

出现此类问题的原因通常是由于服务器证书中配置的域名和客户端请求的域名不一致所导致的。

有两种解决方案：

(1) 重新生成服务器的证书，用真实的域名信息;

(2) 自定义HostnameVerifier,在握手期间，如果URL的主机名和服务器的标识主机名不匹配，则验证机制可以回调此接口的实现程序来确定是否应该允许此连接。可以通过自定义HostnameVerifier实现一个白名单的功能。

代码如下：

HostnameVerifier DO_NOT_VERIFY = new HostnameVerifier() { 
  @Override 
  public boolean verify(String hostname, SSLSession session) { 
    // 设置接受的域名集合 
    if (hostname.equals(...))  { 
         return true; 
    } 
  } 
}; 

HttpsURLConnection.setDefaultHostnameVerifier(DO_NOT_VERIFY);

(3) 客户端证书验证

SSL支持服务端通过验证客户端的证书来确认客户端的身份。这种技术与TrustManager的特性相似。本文将在第5部分代码实现模块，讲解如何让Android应用支持客户端证书验证的方式。

(4) Android上TLS版本兼容问题

之前在接口联调的过程中，测试那边反馈过一个问题是在Android 4.4以下的系统出现HTTPS请求不成功而在4.4以上的系统上却正常的问题。相应的错误如下：

03-09 09:21:38.427: W/System.err(2496): javax.net.ssl.SSLHandshakeException: javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0xb7fa0620: Failure in SSL library, usually a protocol error 

03-09 09:21:38.427: W/System.err(2496): error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure (external/openssl/ssl/s23_clnt.c:741 0xa90e6990:0x00000000)

按照官方文档的描述，Android系统对SSL协议的版本支持如下：

也就是说，按官方的文档显示，在API 16+以上，TLS1.1和TLS1.2是默认开启的。但是实际上在API 20+以上才默认开启，4.4以下的版本是无法使用TLS1.1和TLS 1.2的，这也是Android系统的一个bug。

参照stackoverflow上的一些方式，比较好的一种解决方案如下：

SSLSocketFactory noSSLv3Factory; 
if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.KITKAT) { 
    noSSLv3Factory = new TLSSocketFactory(mSSLContext.getSSLSocket().getSocketFactory()); 
} else { 
    noSSLv3Factory = mSSLContext.getSSLSocket().getSocketFactory(); 
}

对于4.4以下的系统，使用自定义的TLSSocketFactory，开启对TLS1.1和TLS1.2的支持，核心代码：

public class TLSSocketFactory extends SSLSocketFactory { 

    private SSLSocketFactory internalSSLSocketFactory; 

    public TLSSocketFactory() throws KeyManagementException, NoSuchAlgorithmException { 
        SSLContext context = SSLContext.getInstance("TLS"); 
        context.init(null, null, null); 
        internalSSLSocketFactory = context.getSocketFactory(); 
    } 

    public TLSSocketFactory(SSLSocketFactory delegate) throws KeyManagementException, NoSuchAlgorithmException { 
        internalSSLSocketFactory = delegate; 
    } 

    ...... 

    @Override 
    public Socket createSocket(InetAddress address, int port, InetAddress localAddress, int localPort) throws IOException { 
        return enableTLSOnSocket(internalSSLSocketFactory.createSocket(address, port, localAddress, localPort)); 
    } 

    // 开启对TLS1.1和TLS1.2的支持 
    private Socket enableTLSOnSocket(Socket socket) { 
        if(socket != null && (socket instanceof SSLSocket)) { 
            ((SSLSocket)socket).setEnabledProtocols(new String[] {"TLSv1.1", "TLSv1.2"}); 
        } 
        return socket; 
    } 
}

5.代码实现

本部分主要基于第四部分提出的Android应用中使用HTTPS遇到的一些常见的问题，给出一个比较系统的解决方案。

(1) 整体结构

不管是使用自签名证书，还是采取客户端身份验证，核心都是创建一个自己的KeyStore,然后使用这个KeyStore创建一个自定义的SSLContext。整体类图如下：

类图中的MySSLContext可以应用在HttpURLConnection的方式与服务端连接的过程中:

if (JarConfig.__self_signed_https) { 
    SSLContextByTrustAll mSSLContextByTrustAll = new SSLContextByTrustAll(); 
    MySSLContext mSSLContext = new MySSLContext(mSSLContextByTrustAll); 
   SSLSocketFactory noSSLv3Factory; 
   if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.KITKAT) { 
        noSSLv3Factory = new TLSSocketFactory(mSSLContext.getSSLSocket().getSocketFactory()); 
    } else { 
        noSSLv3Factory = mSSLContext.getSSLSocket().getSocketFactory(); 
    } 

    httpsURLConnection.setSSLSocketFactory(noSSLv3Factory); 
    httpsURLConnection.setHostnameVerifier(MY_DOMAIN_VERIFY); 
}else { 
    httpsURLConnection.setSSLSocketFactory((SSLSocketFactory) SSLSocketFactory.getDefault()); 
    httpsURLConnection.setHostnameVerifier(DO_NOT_VERIFY); 
}

核心是通过httpsURLConnection.setSSLSocketFactory使用自定义的校验逻辑。整体设计上使用策略模式决定采用哪种验证机制：

• makeContextWithCilentAndServer 单向验证方式(自定义信任的证书集合)
• makeContextWithServer 双向验证方式(自定义信任的证书集合，并使用客户端证书)
• makeContextToTrustAll (信任所有的CA证书，不安全，仅供测试阶段使用)

(2) 单向验证并自定义信任的证书集合

在App中，把服务端证书放到资源文件下(通常是asset目录下，因为证书对于每一个用户来说都是相同的，并且也不会经常发生改变)，但是也可以放在设备的外部存储上。

public class SSLContextWithServer implements GetSSLSocket { 

    // 在这里进行服务器正式的名称的配置 
    private String[] serverCertificateNames = {"serverCertificateNames1" ,"serverCertificateNames2"}; 

    @Override 
    public SSLContext getSSLSocket() { 
        String[] caCertString = new String[serverCertificateNames.length]; 
        for(int i = 0 ; i < serverCertificateNames.length ; i++) { 
            try { 
                caCertString[i] = readCaCert(serverCertificateNames[i]); 
            } catch(Exception e) { 

            } 
        } 
        SSLContext mSSLContext = null; 
        try { 
            mSSLContext = SSLContextFactory.getInstance().makeContextWithServer(caCertString); 
        } catch(Exception e) { 

        } 
        return mSSLContext; 
    }

serverCertificateNames中定义了App所信任的证书名称(这些证书文件必须要放在指定的文件路径下，并其要保证名称相同)，而后就可以加载服务端证书链到keystore，通过获取到的可信任并带有服务端证书的keystore,就可以用它来初始化自定义的SSLContext了：

@Override 
    public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException { 
        try { 
            originalX509TrustManager.checkServerTrusted(chain, authType); 
        } catch(CertificateException originalException) { 
            try { 
                X509Certificate[] reorderedChain = reorderCertificateChain(chain); 
                CertPathValidator validator = CertPathValidator.getInstance("PKIX"); 
                CertificateFactory factory = CertificateFactory.getInstance("X509"); 
                CertPath certPath = factory.generateCertPath(Arrays.asList(reorderedChain)); 
                PKIXParameters params = new PKIXParameters(trustStore); 
                params.setRevocationEnabled(false); 
                validator.validate(certPath, params); 
            } catch(Exception ex) { 
                throw originalException; 
            } 
        } 
    }

(3) 跳过证书校验过程

和上面的过程类似，只不过这里提供的TrustManager不需要提供信任的证书集合，默认接受任意客户端证书即可：

public class AcceptAllTrustManager implements X509TrustManager { 

    @Override 
    public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException { 
        //do nothing，接受任意客户端证书 
    } 

    @Override 
    public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException { 
        //do nothing，接受任意服务端证书 
    } 

    @Override 
    public X509Certificate[] getAcceptedIssuers() { 
        return null; 
    }

而后构造相应的SSLContext:

public SSLContext makeContextToTrustAll() throws Exception { 
        AcceptAllTrustManager tm = new AcceptAllTrustManager(); 
        SSLContext sslContext = SSLContext.getInstance("TLS"); 
        sslContext.init(null, new TrustManager[] { tm }, null); 

        return sslContext; 
}

The above is the detailed content of Analyze the principles of HTTPS and its use in Android. For more information, please follow other related articles on the PHP Chinese website!