And ye are puffed up, and have not rather mourned, that he that hath done this deed might be taken away from among you [1 Cor. 5:2].

The apostle is using strong language here. He is dealing with a very grievous sin. The congregation in Corinth was compromising with this evil.
We need to recognize that flagrant sin in the church must be dealt with. The Lord Jesus had given detailed instructions in Matthew 18: “Moreover if thy brother shall trespass against thee, go and tell him his fault between thee and him alone: if he shall hear thee, thou hast gained thy brother. But if he will not hear thee, then take with thee one or two more, that in the mouth of two or three witnesses every word may be established. And if he shall neglect to hear them, tell it unto the church: but if he neglect to hear the church, let him be unto thee as an heathen man and a publican” (Matt. 18:15–17).
They did not carry out this procedure in Corinth. This was a case of compromise with evil. John Morley has said that compromise is the most immoral word in the English language. I think I would agree to that. The church in Corinth was compromising itself by compromising with this evil.
There are certain things about this case that we need to note. This case was an acknowledged situation which had no need of proof. This was not a matter of gossip or of hearsay. Paul would never have brought up something like this if it had simply been a rumor.
Also we need to note that it was not a questionable sin. It was a glaring sin, and it was actually recognized by the world outside as being sin. It was incest. This is in contrast to questionable activities, which should not be brought out in the open and dealt with by the church. Let me give an illustration of what I mean.
A lady was converted in the church where I served as pastor. She called me one day about three months after her conversion, and she was very disturbed. She said, “I’m very disappointed and very discouraged. I have been a chain smoker and have wanted to give up cigarettes. I have tried for three months and I have failed. I have come to the place where I hate them and I hate myself for not being able to give them up. What should I do?” I gave her several suggestions. I said, “Look, it is a questionable sin, and it is one that you hate and want to give up. I don’t blame you; your testimony is involved. First of all, continue to pray, and ask your personal friends to continue to pray for you, as you say they are doing. Also I will pray for you. I know God will give you the victory because you want it. Secondly, don’t be discouraged. And the third thing is: please do not tell it to the dear saints in the church. If you do, they will absolutely skin you alive because they consider it the worst sin in the world.” After about three months I saw her coming into the church, and I could tell by her face something had happened. After the service she could hardly wait to talk to me. She said, “I have wonderful news for you. From the day I talked to you down to the present, I haven’t smoked once. God has given me deliverance!”
Now, smoking is one of the things I classify as a questionable sin. It is not mentioned in the Word of God; nor does it have any question of immorality connected with it. Therefore, it is to be handled differently. It is not to be brought before the church for judgment. By contrast, this case of immorality in the Corinthian church was a flaunting of God’s law. Therefore, this needed to be handled with church discipline. There was no doubt about this being a sin. It is not a questionable matter. It was such a horrible sin that it was not even practiced by the Gentiles outside the church.
I would like to say something to our present generation. Living together without being married is sin in God’s sight. It makes no difference what public opinion says about it or how many people are practicing it. The Word of God calls this sin, and there is no other way one can look at it. It is not a questionable sin as far as the Word of God is concerned.
The church in Corinth did not need to establish the fact that the man was living in sin. Their error was that they tolerated it. They condoned the sin by doing nothing about it. They compromised, and that is the worst thing they could have done. You can put this down as an axiom: A pure church is a powerful church; an impure church is a paralyzed church. You can look around you at churches today and see whether or not that is true. McGee, J. V. (1991). Thru the Bible commentary: The Epistles (1 Corinthians) (electronic ed., Vol. 44, pp. 58–60). Nashville: Thomas Nelson. 

1 Co 5:12, 13 outside. Paul never intended himself or the church to be judges of unbelievers outside the church, but to judge those inside (cf. 1 Pet. 4:17). Those on the outside are for God to judge and believers to evangelize. Those who sin on the inside, the church is to put out. Verse 13 is quoted from Deut. 17:7. MacArthur, J., Jr. (Ed.). (1997). The MacArthur Study Bible (electronic ed., p. 1735). Nashville, TN: Word Pub.

Also see:

Mt 18:  Church Discipline