What is conviction of sin by the Holy Spirit?
Jesus speaks of the Jews convicting (convincing) Him of sin in the context of contrasting His truth versus the lie of Satan. Satan lied about death just as others lie about other doctrines of His word. Does the Holy Spirit do the same conviction of the world (convincing) that Jesus speaks of?
John 8:45-47 ASV
(45) But because I say the truth, ye believe me not.
(46) Which of you convicteth me of sin? If I say truth, why do ye not believe me?
(47) He that is of God heareth the words of God: for this cause ye hear them not, because ye are not of God.
John 16:8-11 ASV
(8) And he, when he is come, will convict the world in respect of sin, and of righteousness, and of judgment: (9) of sin, because they believe not on me; (10) of righteousness, because I go to the Father, and ye behold me no more; (11) of judgment, because the prince of this world hath been judged.
The Holy Spirit convicts the world because of their sin of unbelief, of unrighteousness, leading to judgment with Satan in the lake of fire
If the Holy Spirit convicts a human being of sin, righteousness, and judgment, it does not look good because there is no judgment for those who are in Jesus. The conviction should bring the human being to repentance to avoid the judgment with Satan, if the conviction of the Holly Spirit was to transform the world.
Note that Jesus says that the Holy Spirit will convict (convince?) the world of sin because the word does not believe in Him. Jesus does not say that the Holy Spirit will convince the world to believe in Him. The Holy Spirit will convict the world or righteousness because Jesus did go back to God Father when He was risen, even though the world does not believe it. The Holy Spirit convicts of judgment because Satan has been already judged. The classic meaning of the Greek word for conviction (shame, disgrace) is the one to understand because the Holy Spirit will not convince the world to repent and save the world, since the world is destined to judgment with Satan.
The conviction does nevertheless prove or show to the world that they have been wrong about Jesus and His offer of salvation all along and that He is indeed Jehovah God made flesh.
In the end, the message of repentance is being proclaimed, we just have to believe it as it is or to not believe it. If we believe the true message, we must understand it calls on us to become holy obedient Jews, inwardly, according to Galatians 2:14 and Romans 2:29.
The Verdict: all men will believe and confess that Jesus is Jehovah God, upon His second coming
Wherefore I make known unto you, that no man speaking in the Spirit of God saith, Jesus is anathema; and no man can say, Jesus is Lord [YAHWEH], but in the Holy Spirit. 1 Corinthians 12:3 ASV