You have a billion urls, where each has a huge page. How do you detect the duplicate documents?
Suggest a cryptographic hash function, SHA1 or MD5. They are expensive but very good hash function.
Subscribe to:
Post Comments (Atom)
DFS the life without backtracking
Brief Description
This site just records whatever technical questions which I encounter in study. Any discussion and comments are welcome.
Labels
- fundamental
- algorithm
- coding
- array
- object oriented
- brainteaser
- bit operation
- linked list
- hash table
- programming
- bianry tree
- binary tree
- others
- math
- probability
- test
- unix
- binary search
- string
- dynamic programming
- hash
- java
- suffix tree
- longest common substring
- min heap
- prefix sum
- recursion
- binary search tree
- catalan number
- design pattern
- kd tree
- process
- regular expression
- selection algorithm
- stack
- xor
- OS
- backtracking
- dfs
- fibonacci number
- heap
- merge sort
- multithread
- prefix tree
- prime
- priority heap
- quick sort
- semaphore
- sizeof
- string match
- thread
- try catch
- 2 dimension array
- 3 sum
- A* Search
- B+ tree
- DMA
- LCA
- MMU
- Majority vote
- PCB
- STL
- abstract factory pattern
- algorithm list
- arithmetic expression to a binary tree
- bfs
- bit vextor
- compiler
- concurrency
- context switch
- data structure
- deak lock
- destructor
- draw circle
- edit distance
- front end back end
- function pointer
- garbage collection
- gcd
- graph
- heap corruption
- inline
- inode
- interval tree
- inverted index
- longest common subsequence
- longest increasing substring
- map
- marcos
- max heap
- median
- memory aligment
- merge
- mutable
- overload
- page table
- partition
- permutation
- placement new
- return by value
- scheduler
- shift
- shuffle
- signature
- smart pointer
- spell suggestion
- static
- stringstream
- subset sum
- synchronization
- template
- tournament algorithm
- tree
- tree rotation
- virtual table
- volatile
- web crawler
No comments:
Post a Comment