[I2S] Predicor.py error를 해결하기 위한 과정.

2020-09-04 11:38:20.802072: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-09-04 11:38:23.400467: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1 2020-09-04 11:38:23.457175: E tensorflow/stream_executor/cuda/cuda_driver.cc:314] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected 2020-09-04 11:38:23.457270: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (9badf6d81adc): /proc/driver/nvidia/version does not exist 2020-09-04 11:38:23.486693: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2200000000 Hz 2020-09-04 11:38:23.486990: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x204ca00 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-09-04 11:38:23.487025: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/inception_v3/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5 87916544/87910968 [==============================] - 2s 0us/step Traceback (most recent call last): File "Predictor.py", line 96, in <module> main() File "Predictor.py", line 41, in main result = predictor(args.input, encoder=encoder, decoder=decoder, optimizer=optimizer, tokenizer=tokenizer) File "Predictor.py", line 61, in predictor result = evaluate(path, encoder=encoder, decoder=decoder, tokenizer=tokenizer) File "Predictor.py", line 82, in evaluate predictions, hidden, attention_weights = decoder(dec_input, features, hidden) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/content/MyDrive/My Drive/setup/Predictions/I2S_Model.py", line 46, in call x = self.embedding(x) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 982, in __call__ self._maybe_build(inputs) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 2643, in _maybe_build self.build(input_shapes) # pylint:disable=not-callable File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/utils/tf_utils.py", line 323, in wrapper output_shape = fn(instance, input_shape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/embeddings.py", line 149, in build constraint=self.embeddings_constraint) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 614, in add_weight caching_device=caching_device) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/base.py", line 731, in _add_variable_with_custom_getter name=name, shape=shape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/base.py", line 798, in _preload_simple_restoration checkpoint_position=checkpoint_position, shape=shape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/base.py", line 75, in __init__ self.wrapped_value.set_shape(shape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1209, in set_shape (self.shape, shape)) ValueError: Tensor's shape (10005, 600) is not compatible with supplied shape (69, 600) WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.embedding.embeddings WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.gru.state_spec WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.fc1.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.fc1.bias WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.fc2.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.fc2.bias WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.gru.cell.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.gru.cell.recurrent_kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.gru.cell.bias WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.embedding.embeddings WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.fc1.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.fc1.bias WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.fc2.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.fc2.bias WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.gru.cell.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.gru.cell.recurrent_kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.gru.cell.bias WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.embedding.embeddings WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.fc1.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.fc1.bias WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.fc2.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.fc2.bias WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.gru.cell.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.gru.cell.recurrent_kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.gru.cell.bias WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.

 

 

자료가 날라갔는데.. 다시 정리해보면..

smiles를 벡터화 시키지 못했다. 

smiles를 어떻게 token화 시킬 것인가?

 

https://pypi.org/project/SmilesPE/

SmilesPE

https://github.com/EBjerrum/SMILES-enumeration

EBjerrum/SMILES-enumeration

둘중에 어떤것을 써 볼것인가?

첫번째것을 일단 써보자. 

그리고 논문에서는 어떻게 smiles로  translate back했는지 알아보자.

smilesPE를 이용해서 smile을 token화 하는 과정을 추가하고 기존 코드 그대로 keras의 Tokenizer를 그대로 이용하였다. 

	spe_vob= codecs.open('/content/MyDrive/My Drive/setup/Image2SMILES/SPE_ChEMBL.txt')
	spe = SPE_Tokenizer(spe_vob)
	tkd_train_captions = []
	for smile in train_captions : 
		smile_tkd = spe.tokenize(smile)
		tkd_train_captions.append('<start> ' + smile_tkd +' <end>')

	max_voc = 10000

	#Creating tokenizer with defined characters
	tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=max_voc,oov_token="<unk>",filters='!"$&:;?^`{}~ ',lower=False)
	tokenizer.fit_on_texts(train_captions)
	train_seqs = tokenizer.texts_to_sequences(train_captions)

	tokenizer.word_index['<pad>'] = 0
	tokenizer.index_word[0] = '<pad>'

	cap_vector = tf.keras.preprocessing.sequence.pad_sequences(train_seqs, padding='post')
	max_length = calc_max_length(train_seqs)
		

과연 학습이 잘 되고. 에러가 뜨지 않을지 궁금하다. 지금 학습중..

tokenizer가 뭔지 알아야 했으므로 다음 사이트들을 참고하였다. 

https://subinium.github.io/Keras-6-1/

알고보니 인코딩이 또 이렇게 되어있었다...왜지?

train_captions = tkd_train_captions --> 이렇게 바꿔줬어야하는데 바꿔주질 않아서..

 

실행을 했는데 왜 되다 마는 것일까...

아니!! 에러가 뜨는것도 아니고 왜 실행이 안되는거냐고!

 

왜 두개가 차이가 나는 것인가? 

	#Splitting the text file and saving SMILES as Captions
	for line in smiles.split('\n'):
		tokens = line.split(',')
		image_id = str(tokens[0])+'.png'
		try:
			caption = '<start> ' + str(tokens[1].rstrip()) + ' <end>'
		except IndexError as e:
			print (e,flush=True)
		full_image_path = PATH + image_id

		all_img_name.append(full_image_path)
		all_smiles.append(caption)

이부분에서 잘못 되었다. deepsmiles에서 <strat>  <end>를 우선 빼주고 token으로 나눈 다음에 추가 했어야한다.

오케이 제대로 되었다.

깨달았다!! 이미 5 epoch을 한 상태였음. 그래서 안되었던 것임.

start_epoch = 0
#Loading checkpoint to last saved
if ckpt_manager.latest_checkpoint:
	ckpt.restore(tf.train.latest_checkpoint(checkpoint_path))
	start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])

tensorflow.python.framework.errors_impl.UnknownError: OSError: [Errno 5] Input/output error: '/content/MyDrive/My Drive/setup/Data/train_npy/train_339.png.npy' Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 244, in __call__ ret = func(*args) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 302, in wrapper return func(*args, **kwargs) File "./Image2SMILES/Trainer_Image2Smiles.py", line 63, in map_func img_tensor = np.load(img_name.decode('utf-8').replace('/Train_Images', '/train_npy/')+'.npy') File "/usr/local/lib/python3.6/dist-packages/numpy/lib/npyio.py", line 428, in load fid = open(os_fspath(file), "rb") OSError: [Errno 5] Input/output error: '/content/MyDrive/My Drive/setup/Data/train_npy/train_339.png.npy' [[{{node PyFunc}}]] ^

 

또 에러가 떳나.. 그래서 그냥  ipynb로 하나하나 실행했더니 학습이 잘 되고 있다.

 

5epoch학습결과.

ValueError: Tensor's shape (399, 600) is not compatible with supplied shape (69, 600)

 

또 안된다. ㅜㅜ

--> tokenizer일 때문일 가능성.

--> tokenizer를 불러와서 학습해 봐야겠다.

저장되있던 tokenizer를 확인 해 보아야 겠다. -->코드 상에 있는 것과 똑같은 tokenizer이다..

--> tokenizer를 저장한 이유는. predictor에서 쓰기 위함.