删除重复的行而不进行排序

本文介绍了删除重复的行而不进行排序的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我在Python中有一个实用程序脚本:

I have a utility script in Python:

#!/usr/bin/env python import sys unique_lines = [] duplicate_lines = [] for line in sys.stdin: if line in unique_lines: duplicate_lines.append(line) else: unique_lines.append(line) sys.stdout.write(line) # optionally do something with duplicate_lines

这个简单的功能(uniq，无需先排序，保持稳定的顺序)必须作为一个简单的UNIX实用程序提供，不是吗?也许是管道中的过滤器组合?

This simple functionality (uniq without needing to sort first, stable ordering) must be available as a simple UNIX utility, mustn't it? Maybe a combination of filters in a pipe?

问的原因:在无法从任何地方执行Python的系统上需要此功能.

Reason for asking: needing this functionality on a system on which I cannot execute Python from anywhere.

推荐答案

UNIX Bash脚本博客建议:

The UNIX Bash Scripting blog suggests:

awk '!x[$0]++'

此命令告诉awk要打印哪些行.变量$0包含一行的全部内容，方括号可用于数组访问.因此，对于文件的每一行，如果先前未设置该节点的内容(!)，则数组x的节点将递增，并打印该行.

This command is telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array x is incremented and the line printed if the content of that node was not (!) previously set.